SuperConText: Supervised Contrastive Learning Framework for Textual Representations

In the last decade, Deep neural networks (DNNs) have been proven to outperform conventional machine learning models in supervised tasks. Most of these are typically optimized by minimizing well-known Cross-Entropy objective function. The latter, however, has a number drawbacks, including poor margins and instability. Taking inspiration from recent self-supervised Contrastive representation approaches, we introduce Supervised framework for Textual representations (SuperConText) address those issues.We pretrain network novel fully-supervised contrastive loss. goal is increase both inter-class separability intra-class compactness embeddings latent space. Examples belonging same class regarded as positive pairs, while examples different classes considered negatives. Further, propose simple yet effective method selecting hard negatives during training phase. an extensive series experiments, study impact parameters on quality learned (e.g. batch size). Simulation results show that proposed solution outperforms several competing approaches various large-scale text classification benchmarks without requiring specialized architectures, data augmentations, memory banks, or additional unsupervised data. For instance, achieve top-1 accuracy 61.94% Amazon-F dataset, which 3.54% above best result obtained when using cross-entropy with model architecture.